Assignment 2: Distributions

Essay

Normal Distribution

\[ f(x)=\frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) \] The normal distribution is one of the most fundamental distributions in statistics. It has a bell-shaped curve and is perfectly symmetric about its center. It is unimodal, meaning it has only one peak. In a normal distribution, the mean, median, and mode are all equal and are located at the center, with 50% of the observations falling to the left and 50% to the right.

Any normal distribution can be standardized into a standard normal distribution, which has a mean of 0 and a standard deviation of 1. This makes it especially useful for probability calculations and statistical analysis. According to the Empirical Rule, approximately 68% of observations fall within one standard deviation of the mean, about 95% within two standard deviations, and about 99.7% within three standard deviations.

The total area under the normal curve is equal to 1, representing the total probability. The distribution is continuous and extends from negative infinity to positive infinity. It is fully described by two parameters: the mean, which determines the center, and the standard deviation, which determines the spread. \[ X \sim N(\mu, \sigma^2) \] If the population is normally distributed, then the distribution of the sample mean is also normal. Even better, the Central Limit Theorem states that even if the population distribution is unknown, as long as the sample size is large enough, the distribution of the sample mean will be approximately normal. This is very helpful for z-tests, confidence intervals, and hypothesis tests about the mean. \[ \bar{X} \sim N\!\left(\mu,\; \frac{\sigma^2}{n}\right) \]

T-distribution

The t-distribution is very similar in shape to the standard normal distribution, but it has heavier tails. The standard normal distribution \(N(0, 1)\) is commonly used to obtain z-scores for statistical tests when the population standard deviation is known. However, in most real statistical problems, the population variance is unknown and must be estimated from the sample.

Because this estimation introduces additional uncertainty, the t-distribution accounts for it by having wider tails than the normal distribution. This means that extreme values are more likely under the t-distribution than under the standard normal distribution. As a result, hypothesis tests using the t-distribution are more conservative, making it harder to incorrectly reject the null hypothesis.

x <- seq(-4, 4, length = 400)

y_norm <- dnorm(x)
y_t <- dt(x, df = 5)
y_tt <- dt(x, df = 10)

comparison.df <- data.frame(
  x = rep(x, 3),
  density = c(y_norm, y_t, y_tt),
  distribution = factor(rep(c("Normal", "t (df = 5)", "t (df = 10)"), each = length(x)))
)


t.plt <- ggplot(comparison.df, aes(x = x, y = density, color = distribution)) +
  geom_line(size = 1) +
  labs(title = "t-Distribution vs Normal Distribution",
       x = "Value", y = "Density") +
    theme(plot.title = element_text(hjust = 0.5),
        plot.margin = margin(t = 35, r = 20, b = 30, l = 30, unit = "pt")) +
   scale_color_manual(values = c("red", "blue", "green"))
ggplotly(t.plt)

Although the Central Limit Theorem allows the normal distribution to be used when the sample size is large, this approximation is not always appropriate, especially for small samples. In these situations, the t-distribution is necessary.

The t-distribution is characterized by one parameter called the degrees of freedom (v), which is typically equal to n−1. The degrees of freedom determine how heavy the tails are. As the sample size increases, the degrees of freedom increase, and the t-distribution approaches the standard normal distribution. \[ T \sim t_\nu \] When the population is normally distributed but the population standard deviation is unknown, the standardized sample mean follows a t-distribution instead of a normal distribution. When the sample size is small, this accounts for the extra uncertainty from estimating the standard deviation using the sample. As the sample size increases, the t-distribution becomes closer to the normal distribution. This is very helpful for t-tests and confidence intervals about the mean when standard deviation is unknown. \[ T = \frac{\bar{X} - \mu}{S / \sqrt{n}} \sim t_{\,n-1} \]

Chi-squared distribution

The chi-squared distribution is primarily used for hypothesis testing with categorical data or counts. It is commonly used for goodness-of-fit tests, which examine how well observed data match expected values, and for tests of independence, which determine whether two categorical variables are related. The shape of the chi-squared distribution depends on the number of degrees of freedom, and it becomes more symmetric as the degrees of freedom increase.

In addition, the chi-squared distribution plays an important role when working with normally distributed data. If data comes from a normal population, then after proper scaling, the sample variance follows a chi-squared distribution. Suppose we have k independent standard normal random variables. If we square each one and add them together, the resulting sum, called Q, follows a chi-squared distribution with k degrees of freedom. \[ Q=\sum_{i=1}^k Z_i^2 \sim \chi_k^2. \] This result is important because any normally distributed variable can be converted into a standard normal variable using a z-score. After standardizing the data, we can square these values and sum them, which produces a chi-squared distribution. Because sample variance is based on squared deviations from the mean, it can be standardized in a similar way, and the resulting statistic also follows a chi-squared distribution. \[ \frac{(n-1)S^2}{\sigma^2} \sim \chi_{n-1}^2. \] As a result, we can use the chi-squared distribution to construct confidence intervals for the population variance, test hypotheses about variance, and help derive the t-distribution, which will be discussed in the next section.

F-distribution

Similar to the chi-squared distribution, the F distribution is right-skewed, non-negative, and defined on the interval \([0, \infty)\). Its shape depends on two degrees of freedom. The F distribution is mainly used to compare two sources of variation, especially when comparing explained variation to unexplained variation in regression, or when comparing two population variances.

The F distribution is formed by taking the ratio of two independent sample variances that come from normal populations. Since sample variances follow chi-squared distributions, their properly scaled ratio follows an F distribution. As a result, the F distribution can be used to test whether two population variances are equal. For example, if two classes take the same exam but one class has much less variation in scores than the other, this may suggest that the two classes are not performing under the same conditions. An F-test can be used to determine whether the difference in variances is likely due to random chance or reflects a real difference. \[ F = \frac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2} \sim F_{n_1-1,\; n_2-1} \] In regression and ANOVA, the F statistic is calculated as the ratio of explained variation vs unexplained variation, MSR/MSE. If the model explains little more than random noise, the F value will be close to 1. If the model explains a large amount of variation, the F value will be much larger than 1. This helps you evaluate how good of a fit your model is. Therefore, the F distribution is used to determine whether observed patterns in the data are likely due to real relationships or simply due to random variation. \[ F = \frac{MSR}{MSE} \sim F_{p,\; n-p-1} \] Therefore, the F distribution is used to determine whether observed patterns in the data are likely due to real relationships or simply due to random variation.